The Development of Text-Mining Tools and Algorithms

This paper describes the first version of the TextMOLE (Text Mining Operations Library and Environment) system for textual data mining. Currently TextMOLE acts much like an advanced search engine: it parses a data set, extracts relevant terms, and allows the user to run queries against the data. The system design is open-ended, robust, and flexible. The tool is designed as a utility for quickly analyzing a corpus of documents and determining which parameters will provide maximal retrieval performance. Thus an instructor can use the tool to demonstrate artificial intelligence concepts in the classroom, or use the tool to encourage hands on exploration of the concepts often covered in an introductory course in information retrieval or artificial intelligence. Reseachers will find the tool useful when a ’quick and dirty’ analysis of a unfamiliar collection is required. In addition to discussion of TextMOLE, this paper describes an algorithm that uses TextMOLE as a platform for testing and implementing. The most common retrieval systems run queries independently of one another — no data about the queries is retained from query to query. This paper describes an unsupervised learning algorithm that uses information about previous queries to prune new query results. The algorithm has two distinct phases. The first trains on a batch of queries; in doing so, it learns about the collection and the relationship between its documents. It stores this information in a document-bydocument matrix. The second phase uses the accumulated knowledge to prune query results. Documents are removed from query results based on their learned relationship to documents at the top of the results list. The algorithm can be fine-tuned to be very aggressive or more conservative in its pruning. This algorithm produced increased relevancy of the results and significantly reduces the size of the results list.